Method and device for controlling remote capturing of an image based on detection of a predetermined change in the image

ABSTRACT

A portable telephone is provided with a camera module ( 13 ) for outputting a captured image as image information, a memory ( 105 ) for storing the image information, a face registration unit ( 504 ) for holding information relating to a face image, a face extraction unit ( 501 ), a face parameter extraction unit ( 502 ), and a matching determination unit ( 505 ) which serve as the configuration for detecting the face image held in the face registration unit ( 504 ) from the captured image, and an image-capturing control unit ( 506 ) for executing control processing for image capturing. The image-capturing control unit ( 506 ) stores the image information in the memory ( 105 ) on the basis of the fact that after the face image held in the face registration unit ( 504 ) is detected, the face image becomes undetected, and the face image is then detected again.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the national phase of PCT applicationPCT/JP2009/069870 having an international filing date of 25 Nov. 2009,which claims benefit of Japanese application No. 2008-300761 filed 26Nov. 2008. The contents of the above patent applications areincorporated by reference herein in their entirety.

TECHNICAL FIELD

The present invention is related to devices with camera such as cellularphones with camera, PDAs (Personal Digital Assistant) with camera, anddigital cameras, etc.

BACKGROUND ART

Conventionally, a device with camera normally comprises a self-timerfunction. This self-timer function is an automatic image-capturefunction used when there is nobody to press the shutter. When theshutter button is pressed while the self-timer has been set, the shutterclicks after a set time, and the captured image is saved in a memory.After pressing the shutter button, the user moves to the position beingcaptured, prepares for image capture and waits for the shutter to click.

However, when using the self-timer function, there are time constraints.Consequently, if one is slow to prepare for image capture for somereason, the shutter may click before preparation is complete. In thiscase, it is necessary to reset the self-timer to perform image capture.

At the same time, such devices with camera comprise a control functionprovided by a remote control that enables remote operation using theremote control to click the shutter (i.e., a remote-shutter function)(e.g., Patent Document 1).

When performing image-capture operations using a remote control in thisway, it is possible to click the shutter at the photographer's desiredtiming after sufficiently preparing for image capture.

Furthermore, devices with camera that detect a subject's face and theirfacial expression and automatically click the shutter according tofacial expressions or movements have recently been developed (e.g.,Patent Document 2).

PRIOR ART DOCUMENT Patent Document

-   Patent Document 1: Japanese published unexamined application    2007-41205 [G03B 17/38]-   Patent Document 2: Japanese published unexamined application    2008-42319 [HO4N 5/232]

SUMMARY OF THE INVENTION

The objective of the present invention is to provide a device withcamera that allows a shutter operation to be performed at a desiredtiming and provides excellent operability.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a front view of the external configuration of a cellularphone according to one embodiment.

FIG. 1B is a lateral view of the external configuration of a cellularphone according to one embodiment.

FIG. 1C is a cross-sectional diagram along the line A-A′ of the externalconfiguration of a cellular phone according to one embodiment.

FIG. 2 is a block diagram showing the overall configuration of acellular phone according to an embodiment.

FIG. 3 is a functional block diagram for realizing a remoteimage-capture mode according to one embodiment.

FIG. 4 is a diagram showing the process flow of an image-capture processexecuted in a remote image-capture mode according to one embodiment.

FIG. 5A is an explanatory diagram of the flow in a remote image-capturemode according to one embodiment.

FIG. 5B is an explanatory diagram of shifts in a detected matching rateand related control actions according to one embodiment.

FIG. 6 is a function block diagram for realizing a remote image-capturemode according to one embodiment.

FIG. 7A is a diagram showing the process flow of an image-captureprocess executed in a remote image-capture mode according to oneembodiment.

FIG. 7B is a diagram showing the process flow for performing imagecapture based on the detection results of the presence or absence of theface of a registered user in a remote image-capture mode according toone embodiment.

FIG. 8 is a functional block diagram for realizing a remoteimage-capture mode according to one embodiment.

FIG. 9 is a diagram showing the process flow of an automaticimage-capture process according to one embodiment.

FIG. 10 is a functional block diagram for realizing a remoteimage-capture mode according to one embodiment.

FIG. 11A is a diagram showing the process flow of the automatic settingsin a remote image-capture mode according to one embodiment.

FIG. 11B is a diagram showing the process flow of an automatic releasein a remote image-capture mode according to one embodiment.

FIG. 12A is an explanatory diagram of the automatic settings in a remoteimage-capture mode according to one embodiment.

FIG. 12B is an explanatory diagram of the flow of an automatic releasein a remote image-capture mode according to one embodiment.

FIG. 12C is an explanatory diagram of the flow of an automatic releasein a remote image-capture mode according to one embodiment.

FIG. 13 is a functional block diagram of a case in which theconfiguration is applied to one embodiment.

EXPLANATION OF REFERENCES

-   13: Camera module (camera part)-   105: Memory (memory part)-   501: Face extraction part (face detection part)-   502: Face-parameter extraction part (face detection part)-   504: Face registration part-   505: Matching judgment part (face detection part)-   506: Image-capture control part (face detection part, remote memory    part)-   511: Target identification part (face detection part)-   512: Target-tracking part (face detection part, tracking part)-   513: Face-presence detection part (face detection part)-   521: State detection part-   531: Mode control part

EMBODIMENTS FOR CARRYING OUT THE INVENTION First Practical Example ofthe Present Invention

The following is a description of an embodiment of the present inventionwith reference to the drawings. FIG. 1 is a diagram showing the externalconfiguration of a cellular phone, FIG. 1A is a front view of thecellular phone in a state in which a second cabinet is open, and FIG. 1Bis a lateral view in the same state. FIG. 1C is a cross-sectionaldiagram along the line A-A′ in FIG. 1B.

The cellular phone comprises a first cabinet 1 and the second cabinet 2.A numeric keypad part 11 is arranged on the front-surface side of thefirst cabinet 1. The numeric keypad part 11 comprises multiple numericaland character keys, a call-starting key, and a call-ending key, etc. Abacklight device 12 (hereinafter referred to as “key backlight”) isarranged on the rear of the numeric keypad part 11. The key backlight 12comprises an LED that acts as a light source, and feeds light to thenumeric keypad part 11. As a result, the user is able to see thedisplays attached to each key even when the surroundings are dark.

A camera module 13 is arranged inside the first cabinet 1. The lenswindow (not illustrated) of the camera module 13 is arranged on the rearsurface of the first cabinet 1, and from this lens window, an image ofthe subject is downloaded into the camera module 13.

A display 21 with a vertically long rectangular shape is arranged on thefront-surface side of the second cabinet 2, and the display surfacethereof is arranged on the front surface. This display 21 may beconfigured using, for example, a liquid crystal panel or an organic EL,etc. A backlight device 22 (hereinafter referred to as “panelbacklight”) is arranged on the rear of the display 21. The panelbacklight 22 comprises an LED that acts as a light source, and feedslight to the display 21.

Furthermore, a main-key part 23 is arranged on the front-surface side ofthe second cabinet 2. On the main-key part 23, mode keys for activatingvarious functional modes (Camera mode, E-mail mode, Internet mode,Address-book mode), movement keys for scrolling through a screen ormoving a highlight focus, and an Enter key for performing variousdetermination operations.

The numeric keypad part 11 and the main-key part 23 may be realizedusing a touch panel or a touch sensor.

The second cabinet 2 is connected to the first cabinet tin a mannerallowing it to slide in the X-axis direction in FIG. 1 using a slidemechanism part 3. As shown in FIG. 1C, the slide mechanism 3 comprises aguide board 31 and a guide groove 32. The guide board 31 is provided onboth the left and right ends of the rear surface of the second cabinet2, and comprises a protrusion 31 a on the lower end. The guide groove 32is formed on the lateral surface of the first cabinet 1 along thesliding direction (X-axis direction in FIG. 1). The protrusion 31 a ofthe guide board 31 is engaged with the guide groove 32.

As indicated by the dashed line in FIG. 1B, when the cellular phone isclosed, the second cabinet 2 almost completely overlaps the firstcabinet 1. In this state (closed state), all of the keys on the frontsurface of the numeric keypad part 11 are hidden behind the secondcabinet 2. The second cabinet 2 is able to slide until the guide board31 reaches the end position of the guide groove 32 (i.e., enter an openstate). When the second cabinet 2 is completely open, as shown in FIG.1A, all of the keys of the numeric keypad part 11 are externallyexposed.

FIG. 2 is a block diagram showing the overall configuration of thecellular phone. In addition to each component described above, thecellular phone of the present embodiment comprises a CPU 100, a videoencoder 101, a microphone 102, an audio encoder 103, a communicationmodule 104, a memory 105, a backlight driver circuit 106, a videodecoder 107, an audio decoder 108, and a speaker 109.

The camera module 13 comprises an imaging lens 13 a, and an imagingelement 13 b, etc. The imaging lens 13 a forms an image of a subject onthe imaging element 13 b. The imaging element 13 b comprises, forexample, a CCD, and generates and outputs image signals according to thedownloaded image to the video encoder 101. The video encoder 101converts the image signals from the imaging element 13 b into digitalimage signals that can be processed by the CPU 100, and outputs thesignals to the CPU 100.

The microphone 102 outputs audio signals into electrical signals andoutputs the signals to the audio encoder 103. The audio encoder 103converts the audio signals from the microphone 102 into digital signalsthat can be processed by the CPU 100, and outputs the signals to the CPU100.

The communication module 104 converts the audio signals and imagesignals and text signals, etc. from the CPU 100 into radio signals andsends the signals to a base station via an antenna 104 a. Thecommunication module 104 also converts radio signals received via theantenna 104 a into audio signals, image signals and text signals, etc.and outputs the signals to the CPU 100.

The memory 105 includes a ROM and a RAM. The memory 105 stores a controlprogram for granting control functions to the CPU 100. Moreover, imagedata captured using the camera module 24, image data downloaded from anexternal source via the communication module 104, and text data (E-maildata), etc. are saved in the memory 105 in prescribed file formats.

The backlight driver circuit 106 feeds voltage signals corresponding tocontrol signals from the CPU 100 to the panel backlight 22 and the keybacklight 12.

The video decoder 107 converts video signals from the CPU 100 intoanalog video signals that can be displayed on the display 21, andoutputs the signals to the display 21.

The audio decoder 108 converts audio signals from the CPU 100 intoanalog audio signals that can be output from the speaker 109, andoutputs the signals to the speaker 109. The speaker 109 plays the audiosignals from the audio decoder 108 as audio.

By outputting control signals to parts such as the communication module104, the video decoder 107, and the audio decoder 108, etc. based oninput signals from parts such as the camera module 13, the microphone102, the main-key part 23, and the numeric keypad part 11, etc., the CPU100 performs processes for various functional modes.

For example, when the mode key for Camera mode is pressed, the CPU 100activates the camera. In this Camera mode, images sent from the cameramodule 13 are displayed on the display 21 as preview images. Becauseseveral tens of frames are sent every second from the camera module 13,images are displayed on the display 21 in a video-like manner.

When the user presses a shutter key (assigned to, for example, the Enterkey), the CPU 100 stores the image captured at that timing in the memory105 in a prescribed file format such as JPEG, etc.

The cellular phone of the present embodiment comprises a function for aremote image-capture mode as one function of the Camera mode. When theuser is joining the photograph being captured, the remote image-capturemode can be activated by performing a prescribed key operation.

FIG. 3 is a functional block diagram for realizing the remoteimage-capture mode.

In order to realize the remote image-capture mode, the CPU 100 and thememory 105 comprise a face extraction part 501, a face-parameterextraction part 502, a registration control part 503, a faceregistration part 504, a matching judgment part 505, and animage-capture control part 506.

The face extraction part 501, the face-parameter extraction part 502,the registration control part 503, the matching judgment part 505 andthe image-capture control part 506 are realized as functions of softwareexecuted by the CPU 100. Part of the storage area of the memory 105 isallocated to the face registration part 504.

Image data output from the camera module 13 are input into the CPU 100via the video encoder 101, and these image data are obtained by the faceextraction part 501 and the face-parameter extraction part 502.

The face extraction part 501 obtains image data for one screen (i.e.,for one frame) per one frame or several frames, and based on the imagedata, extracts the region of a face image (face region) contained in theimage. For example, by detecting skin color within an image as well ascharacteristic parts such as the eyes, nose and mouth based on theshading, etc. of the image, the face extraction part 501 extracts a faceregion. Then, it outputs information for identifying the face region(e.g., positional information of the outline of the face) (hereinafterreferred to as “face-region information”) to the face-parameterextraction part 502. If there are multiple face images within a singleimage, the face extraction part 501 extracts all of the face regions.

At the same timing as the face extraction part 501, the face-parameterextraction part 502 obtains image data for one screen, and based on theface-region information sent from the face extraction part 501, extractsparameters for distinguishing the face image from the image datacorresponding to the face region. Examples of parameters include therespective shapes of parts such as the eyes, nose and mouth, the ratiosof the distances between each part, and skin color, etc.

In this way, the face-parameter extraction part 502 extracts parametersof each face image for all of the face regions that have been extractedby the face extraction part 501, and outputs the parameters to thematching judgment part 505.

The matching judgment part 505 cross-checks the face-image parametersthat have been sent with the face-image parameters registered in theface registration part 504, and calculates the matching rate(conformance rate) with the registered face image.

Registration of face-image parameters in the face registration part 504is performed by the registration control part 503. After receiving auser input of a registration operation, the registration control part503 instructs the face-parameter extraction part 502 to send parametersto the face registration part 504. After receiving this instruction, theface-parameter extraction part 502 sends the face-image parametersextracted at that moment to the face registration part 504. In this way,face-image parameters are registered in the face registration part 504.

The matching judgment part 505 calculates the matching rate of all ofthe extracted face regions with respect to the face images and outputsthe results to the image-capture control part 506.

By comparing the sent matching rate to a preset threshold value, theimage-capture control part 506 determines whether a registered faceimage is present in the image, and performs an image-capture processbased on the determined result.

FIG. 4 is a diagram showing the process flow of an image-capture processexecuted in a remote image-capture mode. FIG. 5A is a diagram usingillustrated images to show the operational flow from the activation ofthe remote image-capture mode to the completion of image capture. FIG.5B is a diagram showing shifts in the detected matching rate and theflow of related control actions. In the bar graph of the matching rateshown in FIG. 5B, as a matter of convenience, the bars for the faceimages of the same person are portrayed as being constantly at the sameheight, but because the matching rate varies even for the face image ofthe same person due to slight changes such as changes in orientation,the height of the bars would actually differ slightly. The following isa description of the control actions with reference to these drawings.

First, before using the remote image-capture mode, the user performsoperations to register their own face image. While the user is capturedby the camera module 13, when the user or another person performsregistration operations, the registration control part 503 operates asdescribed above, and the face-image parameters of the user areregistered in the face registration part 504. Consequently, the user isable to set the remote image-capture mode.

Furthermore, instead of performing image capture during registration asdescribed above, it is also possible to use a configuration in whichface-image parameters are extracted from an image that has beenpreliminarily captured and stored in the memory 105 and registered inthe face registration part 504.

Next, when the user performs a key operation for remote image-capturemode, the remote image-capture mode is activated.

When the remote image-capture mode is activated, the face extractionpart 501, the face-parameter extraction part 502, and the matchingjudgment part 505 are constantly operating while the mode is active, andthe matching rates of all of the face images in one image are outputfrom the matching judgment part 505 to the image-capture control part506 in a set time interval (interval of one or more frames).

The image-capture control part 506 first resets the remote flag (RFG)and the shutter flag (SFG) as part of initial setting.

Next, the image-capture control part 506 obtains the matching rates ofthe face images in a single image (S102), compares the obtained matchingrates with a threshold value for each screen, and determines whether anyexceed the threshold value (S103).

If the user who has registered their face image (hereinafter referred toas “registered user”) has not entered the image-capture region of thecamera module 13 (screen (A) in FIG. 5A), even if other people are inthe image and their face images have been extracted, the matching ratesare low values (time region (A) in FIG. 5B).

In this case, the image-capture control part 506 determines that thereis no face image with a matching rate that exceeds the threshold value(S103: NO), and because the RFG remains reset (S105: NO), the processreturns to step S102, and the matching rates of the face images of oneimage that are sent next are obtained.

Subsequently, when the registered user enters the image-capture region(screen (B) in FIG. 5A), the face image of the registered user isextracted. Then, the matching rate of that face image becomes a highvalue (time region (B) in FIG. 5B).

As a result, the image-capture control part 506 determines that there isa face image with a matching rate that exceeds the threshold value(S103: YES), and causes the RFG that is in a reset state to enter a setstate (S104:NO→S107). As a result, it enters a state of preparation forremote image capture. Subsequently, the image-capture control part 506returns to the process of step S102, and obtains the matching rates ofthe face images of one image that are sent next.

After preparing for image capture, the registered user who has reachedthe image-capture position performs an operation to perform imagecapture. Specifically, as a signal for an instruction for image capture,the user performs a movement to hide their face once and then reveal itagain. For example, the registered user performs a movement to lowertheir face once and then returning to the initial position (screens(C)→(D) in FIG. 5A). When the registered user performs a movement tohide their face, because the face image is no longer extracted, there isno longer a face image with a matching rate that exceeds the thresholdvalue, and when the user performs a movement to reveal their face again,the face image with a matching rate that exceeds the threshold value isrevealed again (time regions (C)→(D) in FIG. 5B).

In the state of preparation for remote image capture, when theimage-capture control part 506 determines that there is no face imagewith a matching rate that exceeds the threshold value (S103: NO),because the RFG has been set (S105: YES) and the SFG remains in thereset state (S108: NO), the SFG is set (S109).

Subsequently, when the image-capture control part 506 again determinesthat there is a face image with a matching rate that exceeds thethreshold value (S103: YES), because the RFG and the SFG are both in aset state (S104: YES→S106: YES), automatic image capture is started(S110).

The image-capture control part 506 performs a shutter operation after,for example, a fixed delay time (e.g., 3 seconds) has passed, and causethe image data of the captured image by the camera module 13 at the timeto be stored in the memory 105. At this time, in order to notify theuser, etc. that the image will be taken, a notification sound such as“beep, beep, beep, click” may be output from the speaker 109. Inconjunction with the notification sound, or instead of the notificationsound, it is also possible to cause a light-emitting part such as an LEDto emit light, or to change the display state of the display 21.

Furthermore, the operations for automatic image capture are not limitedto those described above, and it is possible to perform the shutteroperation immediately without waiting for the passage of the above delaytime. If an auto-focus function is provided, it is also possible toperform the shutter operation after adjusting the focus.

In this way, image capture is completed (screen (E) in FIG. 5A). Ifimage capture is to be continued, the registered user performs amovement to hide their face and then reveal it, just as described above.As a result, the image-capture process is executed (step S103: YES→S104:YES→S106: YES→S110), and the image data of the newly captured image arestored in the memory 105.

Subsequently, when the user performs a key operation to release theremote image-capture mode, the remote image-capture mode is stopped.

According to the present embodiment, the registered user is able toperform photography (image storage) at a desired timing that theyindicate simply by performing a movement to hide their own face andreveal it again. Because a remote control is not required, even if oneforgets to bring the remote control, there is no concern that remoteimage capture cannot be performed.

Consequently, it is possible to realize a cellular phone with excellentcamera operability.

Second Practical Example of the Present Invention

In the above embodiment, if a face image with a matching rate exceedingthe threshold value is present in the image at every detection timingpoint, it is deemed that the registered user is present in theimage-capture region. For this reason, when there is a person resemblingthe registered user in the image, even when the face image of theregistered user disappears, if the face image of the resembling personexceeds the threshold value, it is determined that the face image of theregistered user is present. As a result, even if the registered userperforms an image-capture operation, a photograph may not be takenproperly.

One method of dealing with that is to set the threshold value to a highvalue. However, when the image-capture distance of the registered userbecomes greater and the extracted face image becomes smaller, thematching rate decreases. Therefore, if the threshold value is too high,the registered user may not be detected. Moreover, because the matchingrate varies when the registered user simply changes the orientation oftheir face, if the threshold value is too high, this may create asituation in which these variations cause the matching rate to hoverabove and below the threshold value and shutter operations are performedto take photographs unintended by the registered user.

Although the control process becomes complicated compared to the aboveembodiment, in order to accurately determine whether the face image ofthe registered user is present, it is possible to use a configurationsuch as that of Example 1 as described below.

FIG. 6 is a functional block diagram for realizing a remoteimage-capture mode according to Example 1.

In the present example, a target identification part 511, atarget-tracking part 512, and a face-presence detection part 513 arearranged between the matching judgment part 505 and the image-capturecontrol part 506. Other configurations are the same as those of theabove embodiment.

Based on the matching rates obtained from the matching judgment part 505and the face-region information sent from the face extraction part 501,the target identification part 511 identifies which face image of whichface region is estimated to be the face image of the registered user,and outputs the face-region information of that face image to thetarget-tracking part 512.

When receiving the face-region information of the face image of theregistered user, the target-tracking part 512 obtains image data fromthe camera module 13 over a fixed interval (interval of one or moreframes) to track the movements of the target face region, and outputsinformation on the position thereof (hereinafter referred to as“target-position information”) to the face-presence detection part 513.For the target-tracking method, a known method using movement vectors(for example, refer to Japanese published unexamined application2006-323437) may be used, for example.

Based on the face-region information being sent from the face extractionpart 501 and the target-position information being sent from thetarget-tracking part 512, the face-presence detection part 513 detectsthe presence or absence of the face image of the registered user in theimage. If there is face-region information that matches thetarget-position information (i.e., if a face image is detected at theposition of the target being tracked), the face-presence detection part513 assumes that the face image of the registered user is present in theimage and outputs a face-present signal to the image-capture controlpart 506. On the other hand, if there is no face-region information thatmatches the target-position information (i.e., if no face image isdetected at the position of the target being tracked), it assumes thatthe face image of the registered user is not present in the image andoutputs a face-absent signal to the image-capture control part 506.

Based on the result on the presence or absence of the face detected bythe face-presence detection part 513, the image-capture control part 506performs a prescribed image-capture operation.

FIG. 7 is a diagram showing the process flow of an image-capture processexecuted in the remote image-capture mode according to Example 1. FIG.7A is a diagram showing the process for identifying the face image ofthe registered user (target identification process). This targetidentification process is executed by the target identification part 511and the target-tracking part 512. FIG. 7B is a diagram showing theprocess flow of a process for performing image capture based on thedetection results of the presence or absence of the registered user(automatic image-capture process). This automatic image-capture processis executed by the target-tracking part 512, the face-presence detectionpart 513, and the image-capture control part 506. The following is adescription of the control actions with reference to FIG. 7.

When the remote image-capture mode is activated, first, the targetidentification process is performed. The target identification part 511obtains the matching rates of face images and the face-regioninformation for one screen (S201), and compares the obtained matchingrates with a threshold value for each screen to determine whether anyexceed the threshold value (S202). Then, if there is a face image thatexceeds the threshold value (S202: YES), it determines whether thematching rate of that face image is the highest matching rate obtainedsince starting the target identification process (S203). If it is thehighest matching rate (S203: YES), it determines that the face image isthe face image of the registered user, and it sends the face-regioninformation of that face image to the target-tracking part 512. Thetarget-tracking part 512 sets the face region identified by thatface-region information as a target candidate, and starts tracking itsmovements (S204).

The target identification part 511 measures the time from the activationof the remote image-capture mode. In the present example, based on anexpected time between when the registered user performs an operation toset the remote image-capture mode and when the registered user moves toan image-capture position, a search time for ultimately identifying theregistered user is set.

If the search time has not elapsed (S205: NO), the target identificationpart 511 returns to the process of step S201, and obtains the matchingrates of the face images and the face-region information for one screenthat are sent next. In this way, until the search time has elapsed, theprocesses from step S201 to step S204 are repeated. During this time,whenever a face image with the highest matching rate is detected, thetarget candidate is updated, and the movements of the updated targetcandidate are tracked. During this time, if the target candidatedisappears from the image, the target-tracking part 512 discontinuestracking and stands by to receive the next face-region information.

In this way, when the search time has elapsed (S205: YES), the targetidentification part 511 determines whether the target candidate ispresent (S206). Then, if the target candidate is present (S206: YES), itestablishes the current target candidate as the official target (S207)and ends the target identification process. Subsequently, thetarget-tracking part 512 tracks the established target.

In this way, in the present example, by providing a search time andidentifying the face image with the highest matching rate during thattime as the face image of the registered user, even if a personresembling the registered user is in the image-capture region, it ispossible to reduce the identification of the face image of that personas the face image of the registered user.

On the other hand, if the registered user is unable to enter theimage-capture region within the search time due to taking too much timeto move, etc., and a target candidate is not identified within thesearch time (S206: NO), the measurement of the search time is reset(S208), and the process returns to step S201 to repeat operations tosearch for a target candidate.

When the target identification process is completed, the targetidentification part 511 sends a signal indicating that the target hasbeen established to the target-tracking part 512. When receiving thisestablishment signal, the target-tracking part 512 starts the output oftarget-position information to the face-presence detection part 513.

When the target identification process is completed, the automaticimage-capture process is started subsequently. The image-capture controlpart 506 first causes the shutter flag (SFG) to enter a reset state(S301).

The target-tracking part 512 tracks the face image that has beenestablished as the target (i.e., the registered user) (S302), and if thetarget does not leave the image-capture region (S303: NO), thetarget-tracking part 512 outputs the target-position information to theface-presence detection part 513 at the time that the face-regioninformation is output from the face extraction part 501.

As described above, the face-presence detection part 513 detects thepresence or absence of the face image of the registered user based onthe face-region information sent from the face extraction part 501 andthe target-position information sent from the target-tracking part 512,and outputs a face-present signal or a face-absent signal to theimage-capture control part 506 (S304).

As in the above embodiment, when the registered user performs a movementto hide and then reveal their face, the face-presence detection part 513outputs a face-absent signal and then outputs a face-present signal.

When receiving the face-absent signal, the image-capture control part506 determines that the target face image (i.e., the face image of theregistered user) is not present (S305: NO), and because the SFG is in areset state (S307: NO), it sets the SFG (S308). Then, when receiving theface-present signal, it determines that the face image of the registereduser is present (S305: YES), and because the SFG is in a set state(S306: YES), it performs automatic image capture (S309). The operationsfor automatic image capture are the same as those for the aboveembodiment.

If the registered user leaves the image-capture region for some reason,the target-tracking part 512 determines that the target has left theimage-capture region in step S303 (S303: YES). In this case, theautomatic image-capture process is ended, and the target-tracking part512 outputs reset signals to the target identification part 511, theface-presence detection part 513, and the image-capture control part506. Based on this, processes for automatic image capture are restartedfrom the target identification process shown in FIG. 7A.

According to the configuration of the present example, because themovements of a face image determined to be the registered user aretracked, even if a person resembling the registered user is present inthe same image, it is possible to accurately detect the presence orabsence of the face image of the registered user and properly enter theautomatic image-capture process.

Third Practical Example of the Present Invention

When taking a photograph, certain settings and adjustments may beimplemented for image capture depending on each specific situation. Forexample, settings for the use or non-use of the above delay time forautomatic image capture, settings for the use or non-use of a flash, orzoom adjustments, etc. may be implemented. The registered user mayimplement these settings, etc. during or before the implementation ofsettings for remote image-capture mode, but may also do so afterarriving at an image-capture position.

Therefore, the present example has a configuration in which settings,etc. related to image capture can be implemented based on the state ofthe face of the registered user (whether the face is facing right orleft) or the state of other parts (hands, etc.).

FIG. 8 is a functional block diagram for realizing a remoteimage-capture mode according to Example 2. In the present example, astate detection part 521 is added to the functional blocks of Example 1shown in FIG. 6.

Target-position information related to the face image of the registereduser is input into the state detection part 521 from the target-trackingpart 512. Moreover, image data are input into the state detection part521. Based on these inputs, the state detection part 521 detects thestate of the face image of the registered user, and outputs instructionsignals for settings, etc. according to the state to the image-capturecontrol part 506.

In the state detection part 521, states of the face image and thecorresponding instructions are preliminarily set. For example, if theuse or non-use of a delay time is set depending on the horizontalorientation of the face, an instruction for using a delay time is setfor face images facing leftward, and an instruction for not using adelay time is set for face images facing rightward.

For the method of detecting the orientation of the face, a publiclyknown detection method such as the method disclosed in Japanesepublished unexamined application 2000-97676 may be used. In this case,the state detection part 521 detects whether the face image is facingrightward or leftward based on the relative positions of the eyes, noseand mouth, etc. relative to the position of the center of gravity of theskin-color region that is the face region. In other words, if the faceof the registered user faces rightward, because the eyes, nose andmouth, etc. in the skin-color region deviate to the right, the statedetection part 521 determines that the face is facing rightward, andoutputs an instruction signal to use a delay time. On the other hand, ifthe registered user faces leftward, because the eyes, nose and mouth,etc. in the skin-color region deviate to the left, the state detectionpart 521 determines that the face is facing leftward, and outputs aninstruction signal to not use a delay time. Of course, the range ofhorizontal orientation of the face image is the range in which the faceimage can be detected by the face extraction part 501.

Upon receiving an instruction signal for settings, etc. from themovement detection part 521, the image-capture control part 506accordingly performs a process for settings, etc.

FIG. 9 is a diagram showing the process flow of an automaticimage-capture process according to Example 2. In the process flow shownin this diagram, in order to execute settings, etc. related to imagecapture, the processes of the steps S311 and S312 executed by theimage-capture control part 506 are added to the process flow of FIG. 7.

When determining that the face image is present after the face image ofthe registered user (target) has been identified (S305: YES→S306: NO),the image-capture control part 506 determines whether there has been aninstruction for function setting, etc. from the movement detection part521 (S311). Then, if there has been an instruction for function setting,etc., a process complying with that instruction is performed (S312).

For example, if the use or non-use of the above delay time is being set,the use of a delay time is set if the face of the registered user facesrightward, and the non-use of a delay time is set if the face of theregistered user faces leftward.

According to the configuration of the present example, because itbecomes possible for the registered user to implement settings andadjustments related to image capture from a location distant from thecellular phone, the usability improves.

The present example may provide a configuration in which settings, etc.other than the above delay time may be implemented in response to thestate of the face image. For example, if a flash function is provided onthe cellular phone, the use or non-use of the flash may be set, or if azoom function is provided, adjustments for zooming in or out may beperformed. In this case, it is preferable to provide a function forcausing the display 21 to face the same direction as the lens window ofthe camera in order to allow the subject (registered user) to confirmthe zoom conditions.

Moreover, the present example may provide a configuration in whichfunction setting, etc. is implemented in response to the state of partsother than the face, instead of the state of the face. For example, thedirection being pointed to by a finger may be detected, and functionsetting, etc. may be implemented in response to the direction of thefinger. In the case of the present example, because this is aconfiguration in which the face region of the registered user istracked, if the state of parts other than the face is detected, in orderto distinguish whether it is the state of the registered user, it ispreferable to use a configuration in which only the state of thesurroundings of the face image of the registered user is detected.

Fourth Practical Example of the Present Invention

In the above embodiment and Examples 1 and 2, the setting and release ofthe remote image-capture mode are performed through a prescribed keyoperation. Consequently, the registered user performs operations forsetting and release each time they use the remote image-capture mode.Therefore, the present example has a configuration in which the settingand release (activation and stopping) of the remote image-capture modeare performed automatically without relying on a key operation.

FIG. 10 is a functional block diagram for realizing a remoteimage-capture mode according to Example 3. In the present example, amode control part 531 is added to the functional blocks of Example 1shown in FIG. 6.

In Example 1, reset signals from the target-tracking part 512 are outputto the face-presence detection part 513, the image-capture control part506, and the target identification part 511, but in the present example,they are output to the mode control part 531.

Moreover, in the present example, even when the remote image-capturemode is not active, the face extraction part 501, the face-parameterextraction part 502, and the matching judgment part 505 are operating,and in the mode control part 531, matching rates from the matchingjudgment part 505 are input along with face-position information fromthe face extraction part 501.

When the remote image-capture mode is not active, the mode control part531 activates the remote image-capture mode based on the input matchingrates. Moreover, when the remote image-capture mode is active, it stopsthe remote image-capture mode based on the input matching rates,face-region information and reset signals.

FIG. 11 is a diagram showing the process flow for a process forimplementing automatic settings and a process for implementing automaticrelease for a remote-image capture mode according to Example 3. FIG. 11Ashows the process flow for automatic settings, and FIG. 11B shows theprocess flow for automatic release. FIG. 12 is a diagram usingillustrated images to show the flow for automatic settings and releasefor the remote image-capture mode according to Example 3. FIG. 12A showsthe flow for automatic settings, and FIGS. 12B and C show the flow forautomatic release. The following is a description of the control actionswith reference to these drawings.

With reference to FIG. 11A, the mode control part 531 obtains thematching rates of face images for one screen (S401), compares theobtained matching rates with a threshold value for every screen, anddetermines whether any exceed the threshold value (S402).

If the registered user does not enter the image-capture region to jointhe photograph, none of the matching rates exceeds the threshold value(S402: NO), and therefore, the remote image-capture mode remains off(screen (A) in FIG. 12A).

On the other hand, if the registered user enters the image-captureregion to join the image capture, the matching rate exceeds thethreshold value (S402: YES), and therefore, the mode control part 531determines that the face image of the registered user is present in theimage and activates the remote image-capture mode (S403; screen (B) inFIG. 12A).

The present example shifts into a state of preparation for image captureat the same time that the remote image-capture mode is activated. Inother words, unlike Example 1, in the present example, the targetidentification process of FIG. 7A is not performed, the face image thathas been determined as exceeding the threshold value in step S402 (ifthere is more than one, the one with the highest matching rate) isidentified as the face image of the registered user, and the face-regioninformation is output to the target-tracking part 512. As a result, theautomatic image-capture process of FIG. 7B is performed.

Next, with reference to FIG. 11B, while the remote image-capture mode isactive, the mode control part 531 obtains the matching rates andface-region information of the face images for one screen (S501), anddetermines whether the size of the face region of the face imageexceeding the matching rate (i.e., the face image of the registereduser) exceeds a threshold value (has expanded) (S502). If the size ofthe face region does not exceed the threshold value (S502: NO), itdetermines whether the face region (target) has left the image-captureregion (i.e., whether a reset signal has been output from thetarget-tracking part 512) (S503).

While image capture is being continued, because the registered user doesnot move significantly away or leave the image-capture region, the sizeof the face region does not exceed the threshold value (S502: NO), andno reset signal is input (S503: NO). Consequently, in this case, theremote image-capture mode remains active (screen (C) in FIG. 12B, screen(E) in FIG. 12C).

On the other hand, when image capture ends, the registered user facesthe cellular phone (camera module 13). At this time, the face image ofthe registered user comes close to the cellular phone within theimage-capture region or leaves the image-capture region.

The mode control part 531 determines that the size of the face regionhas exceeded the threshold value when the face image comes close to thecellular phone within the image-capture region (S502: YES), and stopsthe remote image-capture mode (S504; screen (D) in FIG. 12B).Alternatively, when a reset signal is input when the face region leavesthe image-capture region (S503: YES), it stops the remote image-capturemode (S504; screen (F) in FIG. 12B).

According to the configuration of the present example, because theremote image-capture mode can be activated and stopped without requiringthe registered user to perform key operations each time, it is possibleto further improve the operability.

The present example has a configuration in which both the activation andstopping of the remote image-capture mode are performed automatically,but a configuration may be provided in which only one is automaticallyperformed.

Moreover, the present example has a configuration in which the remoteimage-capture mode is stopped based on determinations of both when thesize of the face region of the registered user exceeds a threshold valueand when the face region leaves the image, but a configuration may beprovided in which a determination is made only for one.

Furthermore, the present example may be applied to the above embodimentshown in FIG. 3 through FIG. 5. However, because the above embodimentdoes not comprise the target-tracking part 512, the configuration is onein which the remote image-capture mode is stopped only when the size ofthe face region of the registered user exceeds the threshold value.

FIG. 13 is a functional block diagram showing a case in which theconfiguration according to Example 3 is applied to the above embodiment.As shown in this diagram, in the mode control part 531, matching ratesare input from the matching judgment part 505 along with face-regioninformation from the face extraction part 501.

In this configuration, as the process for activating the remoteimage-capture mode, the same process as that shown in FIG. 11 isperformed. On the other hand, for the process for stopping the remoteimage-capture mode, the same process as that shown in FIG. 11B isperformed, but the process of step S503 is excluded. In this case, theprocess flow is one in which, when it is determined that the size of theface region does not exceed the threshold value in step S502, theprocess returns to step S511.

Other

In addition to the above, various changes may be made to the embodimentsof the present invention.

For example, the above embodiment has a configuration in which, imagedata of the captured image are officially recorded in the memory 105immediately when a shutter operation is performed. However, the presentinvention is not limited to this configuration, and may have aconfiguration in which a temporary storage area is provided in thememory 105 or the CPU 100, the captured image is once saved in thistemporary storage area when a shutter operation is performed, and thenofficially stored in the memory 105 when a prescribed save operation isperformed by the registered user, etc.

In this way, the memory part of the present invention described withinthe scope of the patent claims is not limited to a means for officiallystoring captured images, such as the memory 105 of the above embodiment,and also includes means for temporarily storing captured images, such asthe above temporary storage area.

In the above embodiment, an example of a cellular phone with camera wasused, but the device with camera of the present invention is not limitedto a cellular phone, and may be a PDA (Personal Digital Assistant) withcamera or a digital camera, etc.

In addition, the embodiment of the present invention may under varioussuitable changes within the scope of the technical concepts representedin the scope of the patent claims.

Consequently, the photographer is able to perform photography (imagestorage) at a desired timing that they intend. Because a remote controlis not required, even if one forgets to bring the remote control, thereis no concern that remote image capture cannot be performed.

In the device with camera of the present invention, after the face imageis no longer detected after the face image is detected by the facedetection part, the remote memory part executes a control process tocause the memory part to store the image information based on therenewed detection of the face image.

According to this configuration, when a movement to hide and then againreveal the face of the photographer is performed, a shutter operation isperformed based on this movement and the image information is stored.

Consequently, by performing the above movement, the photographer is ableto perform photography (image storage) at a desired timing that theyintend.

The device with camera of the present invention may have a configurationin which the control process of the remote memory part is activated inresponse to the face image being detected by the face detection partafter image capture by the camera part has been started.

By using such a configuration, because the control process for remoteimage capture can be activated without requiring the photographer toperform setting operations each time, the operability is improved.

Furthermore, the device with camera of the present invention may have aconfiguration that further comprises a tracking part that tracks theface image detected by the face detection part in the captured image. Inthis case, the present invention may have a configuration in which,after the control process executed by the remote memory part has beenactivated, the control process executed by the remote memory part isstopped in response to the tracking part detecting that the face imagehas left the captured image.

Alternatively, the device with camera of the present invention may havea configuration in which, after the control process executed by theremote memory part has been activated, the control process executed bythe remote memory part is stopped in response to the face detection partdetecting that the size of the face image has exceeded a thresholdvalue.

By using such a configuration, because it is possible to stop thecontrol process for remote image capture without requiring thephotographer to perform setting operations each time, the operability isimproved.

Furthermore, the device with camera of the present invention may have aconfiguration that further comprises a state detection part that detectsthe kinetic state of a person who is the subject of the face imagedetected in the captured image by the face detection part. In this case,the present invention may have a configuration in which the remotememory part performs function setting related to image capture accordingto the kinetic state.

By using such a configuration, the photographer becomes able to performfunction setting for image capture from a location separated from theequipment.

According to the practical examples of the present invention, it ispossible to provide a device with camera that realizes excellentoperability allowing shutter operations to be performed at a desiredtiming.

The effects and significance of the present invention are made clear inthe descriptions of the embodiment above. However, the followingembodiment is only one example for implementing the present invention,and the present invention is not limited by the following descriptionsof the embodiment.

INDUSTRIAL APPLICABILITY

The present invention is related to a device with camera, and may beused for a device with camera that performs a shutter operation at adesired timing.

What is claimed is:
 1. A device with camera, comprising: a camera partconfigured to output a captured image as image information; a memorypart configured to store the image information and previously registeredface information corresponding to a registrant's face; a face detectionpart configured to detect information regarding a face image from thecaptured image; and an image-capture control part configured to executea control process to cause the memory part to store the imageinformation based on the detection of the face image, wherein thecontrol process comprises a remote image capture mode in which thecaptured face image is compared to the previously registered faceinformation to detect whether the captured face image matches theregistrant's face, and a camera shutter operation is performed inresponse to detecting that the face image matches the registrant's face,and a predetermined change in the captured image has occurred, thecontrol process further comprising, when the face image matches theregistrant's face and a size of the face image detected by the facedetection part becomes greater than a predetermined threshold after theremote image capture mode is started, ending the remote image capturemode.
 2. The device with camera according to claim 1, wherein theimage-capture control part executes the control process to cause thememory part to store the image information based on a renewed detectionof the face image after the face image is no longer detected after theface image is initially detected by the face detection part.
 3. Thedevice with camera according to claim 2, wherein the control processexecuted by the image-capture control part is activated in response tothe detection of the face image by the face detection part after imagecapture by the camera part has been started.
 4. The device with cameraaccording to claim 3, further comprising: a tracking part configured totrack the face image detected by the face detection part, wherein inresponse to the tracking part detecting that the face image is no longerpresent in a subsequently captured image, after the control processexecuted by the image-capture control part has been activated, thecontrol process executed by the image-capture control part is stopped.5. The device with camera according to claim 4, further comprising: astate detection part configured to subsequently detect a motion of aperson who is the subject of the face image detected in the capturedimage by the face detection part, wherein the image-capture control partperforms function setting related to image capture according to thedetected motion.
 6. The device with camera according to claim 2, furthercomprising: a state detection part configured to subsequently detect amotion of a person who is the subject of the face image detected in thecaptured image by the face detection part, wherein the image-capturecontrol part performs function setting related to image captureaccording to the detected motion.
 7. The device with camera according toclaim 3, further comprising: a state detection part configured tosubsequently detect a motion of a person who is the subject of the faceimage detected in the captured image by the face detection part, whereinthe image-capture control part performs function setting related toimage capture according to the detected motion.
 8. The device withcamera according to claim 1, further comprising: a state detection partconfigured to subsequently detect a motion of a person who is thesubject of the face image detected in the captured image by the facedetection part, wherein the image-capture control part performs functionsetting related to image capture according to the detected motion. 9.The device with camera according to claim 1, further comprising: a faceregistration part configured to capture and store face information forone or more registrants in the memory part.
 10. The device of claim 1,wherein if the face image corresponds to the previously registered faceinformation, the remote image capture mode is automatically started evenif the remote image capture mode was not previously set to an ON state.11. The device of claim 1, wherein the previously registered faceinformation and the information regarding the face image include faceparameters obtained from the registrant's face and the face image,respectively.
 12. The device of claim 11, wherein the face image isdetected to be the registrant's face if a matching rate of the faceparameters is not less than a predetermined threshold value.
 13. Adevice with a camera, comprising: a camera part configured to output acaptured image as image information; a memory part configured to storethe image information and previously registered face informationcorresponding to a registrant's face; an image detection part configuredto detect information regarding at least one portion of the capturedimage; and an image-capture control part configured to execute a controlprocess to cause the memory part to store the image information inresponse to detecting that the information regarding the at least oneportion of the captured image matches the registrant's face, whereinwhen the at least one portion of the captured image matches theregistrant's face and a size of the at least one portion detected by theimage detection part becomes greater than a predetermined threshold,after the control process executed by the image-capture control part hasbeen activated and after the image information is stored, the controlprocess executed by the image-capture control part is stopped.
 14. Thedevice of claim 13, wherein if the at least one portion of the capturedimage corresponds to the previously registered face information, thecontrol process is automatically started even if the control process wasnot previously set to an ON state.